Vision system computer modeling apparatus

ABSTRACT

Vision systems for use in modeling environments or scenes are presented. Users view scenes via electronic cameras and are simultaneously presented with CAD type functionality for creating graphical representations of addressed scenes. These systems, having knowledge of position with respect to objects being addressed and with respect to positions measured at prior times, continuously update model perspective and size to reflect a present view of scenes in real time.

BACKGROUND OF THE INVENTION

[0001] This application continues from U.S. application Ser. No.10/350,386 filed Jan. 23, 2003 which continues in part from Ser. No.08/482,944, filed Jun. 7, 1995, presently pending. These inventions aregenerally concerned with computer vision systems and specificallyconcerned with computer vision systems combined with computer aideddesign modeling facilities.

[0002] The following disclosure is presented with the assumption thatelements of computer aided design are well known to the reader. Textsexplaining the current state of that are available from various sources,but those relating to AutoCad® may be particularly helpful. No attemptis made here to explain the details of implementing computer models withsoftware except as it relates to the new arraignment presented.

[0003] Vision systems which combine electronic cameras with computers toperform image functions are described in various texts includingco-pending patent applications of the present inventors having series 08serial numbers: 119,360; 270,911; 355,710; 335,940; 335,912; 411,299;307,360.

[0004] It should be understood that all of the herein referenced provideconsiderable definition of elements of the present invention. Therefore,those materials are incorporated herein by reference whereby the instantspecification can rely upon them for enablement of the particularteachings of each.

[0005] Computer aided design and modeling techniques are useful invarious fields. Commonly referred to as CAD or computer aided design, anelectronic computer can be used to model devices, objects orenvironments. Changes to the model are easily made and a great number ofvariations might be tried before arriving at a final desired result.Examples of uses of CAD include: by architects for buildings proposed tobe built, or by automobile designers for cars proposed to bemanufactured. A designer may start from a conceptual idea, a photograph,an artist drawing or other medium. From that initial idea orrepresentation, a designer may construct on a computer, a model. A modelconsists of a plurality of individual graphical objects each having somecorrespondence to some real object. A photograph suggests to the CADdesigner how particular features of the subject may look and give hintsto how they might be implemented in a graphical representation. Withadvanced computers, it is even possible for the photograph to be scannedinto an electronic image and combined with the CAD drawing, therebyfacilitating formulation of a more accurate model. Converting real worldobjects into a computer model is of great interest to CAD designers asit provides very sophisticated designs in early stages of development.

[0006] Novel techniques have been discovered which provide veryspecialized uses of vision systems, and in particular as they may relateto computer aided modeling and design. While the systems and inventionsof the prior art are designed to achieve particular goals andobjectives, some of those being no less than remarkable, theseinventions have limitations which prevent their use in new ways nowpossible. These prior art inventions are not used and cannot be used torealize the advantages and objectives of the present invention.

SUMMARY OF THE INVENTION

[0007] It is now first presented here, inventions of vision systemcomputer modeling apparatus including devices for creating computermodels of real scenes. It is a primary function of these systems toprovide means and method for computer modeling or real objects. It is acontrast to prior methods and devices that known systems do not interactwith real scenes with respect to perspective and spatial relationshipsas measured in real-time. A fundamental difference between the computermodeling apparatus of these instant inventions and those of the art canbe found when considering its interaction and spatial relationships withrespect to scenes being modeled.

[0008] An electronic imaging system combined with a computer aideddesign system and in communication with position, attitude, andoptionally range measuring devices forms vision system computer modelingapparatus of these inventions. These vision system computer modelingapparatus are operable for addressing real scenes from some user chosenpoint-of-view, or perspective. The model presented at a display as atwo-dimensional image has associated therewith a perspective whichrelates to a perspective of the three-dimensional real scene beingmodeled. The model as presented to a user is continuously responsive tochanges in perspective. When the imaging system is moved such that theperspective of the scene changes, the perspective of the model similarlyand correspondingly changes as well. This is accomplished viameasurements of position and attitude of the apparatus with respect tothe scene being addressed.

[0009] Tools and principles of computer aided design can be combinedwith real time continuous imaging systems to result in sophisticatedmodeling instruments. As it is readily possible to measure spatialinformation including position, attitude and range of the imaging systemwith respect to scenes being modeled, these data can be used to bettercreate advanced models with great accuracy with respect to the realworld.

[0010] For illustration by example, the combination of a computerrunning CAD software; a camcorder electronic camera; and position andattitude measuring apparatus forms the foundation of these systems.

[0011] Using such a system, a model designer may view a scene in realtime and construct a model of the scene as it is viewed. Computergraphical objects can be created and associated with attributes of thereal scene. In particular, the graphic objects and their nodes can beindexed to the real position and orientation of objects and features ofobjects they represent. As the camera is moved about the scene, i.e. toa new perspective, so can the appearance and perspective of thegraphical objects in the computer model in a fashion which correspondsto the way real objects would be imaged from the new camera position. Bycombining computer aided design with real time electronic cameras andposition, attitude and range measuring systems, a novel and powerfulmodeling system is realized.

[0012] These inventions thus stand in contrast to methods and devicesknown previously. The invention is a computer modeling apparatus withlive real-time images and position and attitude measurement deviceswhich relate the computer model to the scene being modeled in properperspective for all relative positions of the apparatus with respect tothe scene. Presently CAD systems are typically computer workstationswith the ability to scan photographs and other graphics which may relateto a scene being modeled. The position and orientation of theworkstation has nothing to do with the models residing within thecomputer.

[0013] It is a primary object of the invention to provide vision systemcomputer modeling apparatus.

[0014] It is additionally an object of the invention to provide computeraided design apparatus whose models are responsive to the position andattitude of the apparatus with respect to the scene being modeled.

[0015] It is an object of the invention to provide apparatus comprisingthe combination of a computer aided design system, an electronic camera,position and attitude measuring apparatus and optionally, rangingapparatus.

[0016] A better understanding can be had with reference to the detaileddescription of Preferred Embodiments and with reference to the appendeddrawings. These embodiments represent particular ways to realize theinvention and are not inclusive of all ways possible. Therefore, theremay exist embodiments that do not deviate from the spirit and scope ofthis disclosure as set forth by the claims, but do not appear here asspecific examples. It will be appreciated that a great plurality ofalternate versions are possible.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] These and other features, aspects, and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims and drawings where:

[0018]FIG. 1 is line drawing of objects having a certain spatialrelationship;

[0019]FIG. 2 is a different view of the same scene;

[0020]FIG. 3 shows a close-up and normal view at once;

[0021]FIG. 4 shows a resizing operation;

[0022]FIG. 5 shows a scene space geometry;

[0023]FIG. 6 shows a top view of that same geometry;

[0024]FIGS. 7 and 8 illustrate line elements generated from variouspositions;

[0025]FIGS. 9, 10, 11, and 12 show the side of a building being modeled;

[0026]FIG. 13 is a flow diagram for a modeling method; and

[0027]FIG. 14 is a system block diagram.

PREFERRED EMBODIMENTS OF THE INVENTION

[0028] In accordance with each of the preferred embodiments of theseinventions, there are provided apparatus for creating a computer modelof a scene where the model is responsive to the position and attitude ofthe apparatus. It will be appreciated that the apparatus of onepreferred embodiment may be different than the apparatus of anotherembodiment. Many alternatives and versions may exist for variouscomponents of these systems. Adoption of certain variations can be madewithout deviation from the true spirit of the invention. The scope ofthe following disclosure should not be limited to the example elementschosen here for illustration. The scope of the invention is set forth bythe appended claims. The following examples are considered by theinventors to be the best of many possible modes of executing theseinventions.

[0029] To realize an apparatus of these inventions, one may combine: 1)a camera; 2) a computer, and 3) position and attitude measuring devices.

[0030] Real-time electronic digital video is available by way of commonCCD type electronic cameras. Hand held camcorders can be battery poweredand highly portable. They allow a user to view some real scene at adisplay device, for example a liquid crystal display device. Inaddition, they are particularly well suited for the present subject asthe digital signal they produce is compatible with computer processingoperations.

[0031] Computers use similar liquid crystal displays for user interface.The images computers generate therefore are compatible with presentationon those devices. Therefore the combination of electronic cameras andcomputers is facilitated by the common display format. Very advanceddesign software is now available having graphical modeling capabilities.CAD methodology permits graphical objects to be manipulated in a greatplurality of ways so that they may represent real objects in a computermodel. These present inventions therefore can most simply be describedas the combination of a real-time electronic camera with a computerrunning CAD type modeling software.

[0032] A user may address some real scene by pointing the camera theretowards from a fixed position. Where a computer is connected to thecamera and is running CAD software, a user may create a model of thereal scene as it is viewed.

[0033] A careful reader will realize that models created with such asystem would necessarily be specific to a certain perspective. For thisreason, the system computer is preferably arranged to be incommunication with position and attitude determining devices. If thedevice is moved, then the perspective of the real scene necessarilychanges. To account for this change in the model, the computermathematically determines a new perspective for the objects of the modelby applying rotation, scale, translation algorithms which are responsiveto the camera's new position such that the perspective of the modelcontinuously updates in accordance with the true perspective of the realscene.

[0034] For example, a camera is addressing a scene, FIG. 1, of a box(11) five meters in front of the camera and a ball (12) five metersfurther in the same direction. If the camera is moved toward the scene,FIG. 2, the objects in the foreground, the box (21), would appear in thedisplay to increase in size at a greater rate than those in thebackground, the ball (22). Given sufficient views of the objects inquestion, the apparatus would have sufficient data to recall a model ofa cube (33) and a sphere (34) to match to the box (31) and the ball(32). The recalled models would then be scaled and oriented by the useror the apparatus itself, and placed so that the models (43, 44)coincided with the real objects (41, 42) as is shown in FIG. 4.

[0035] For the computer to properly size the two objects it may berequired to re-size “stretch” the objects, either manually orautomatically via image recognition for example, to fit the new size ofthe corresponding real box and ball. After such manual re-sizing, thecomputer could estimate via a triangulation routine the relativepositions with respect to the measured camera displacement. Additionalcamera displacements could result in automatic graphic object updates.The first re-sizing operation would calibrate the device and furtheradjustments may be calculated and the re-sizing of the model objectsperformed without further user input. Similarly, other perspective anddepth features could be accounted for. Mathematical translations from areal world 3-D scene to a 2-D computer display are already very common.Many advanced computer routines have been developed to facilitateperspective representations of 3-D objects on a 2-D display. It isimportant to realize here that the invention does not propose any newperspective computation routines. The well developed routines known aresufficient. It is, however, a new combination of primary elements whichresult in the powerful modeling tool.

[0036] An alternate apparatus of the invention may comprise: 1) acamera; 2) a computer; 3) position and attitude measuring devices, and4) a range finding device. The addition of a ranging capability enhancesthe ability of the apparatus to create three dimensional models of thescene being addressed. To calculate the position of a point in thescene, the user would range on the point of interest and the unit, givenposition and attitude data, could calculate the coordinates of the pointdirectly. It is important to realize that the invention does not proposea new method of position calculation based on unit position andattitude, and range to object. This method has been in use, in radardirected gunnery for example, for many years, though not for developingof computer graphical models. What is important is that range basedposition calculation enables the user to quickly develop a model basedon nodes and connecting lines and curves, much as in CAD type programs.Also, range based position calculation enables the unit to use advancedmodeling methods based on image processing such as edge detection withina defined area and coplanar modeling.

[0037] For example, a user of the apparatus may choose to use a methodof coplanar modeling, FIG. 13. A user would range to sufficient points,at least three are required to define a plane but the user could defineas many points as are needed to define the area of interest, to definethe boundaries of a planar area 1302. These boundaries may or may not bepart of the model. The user may also define areas within these edgeboundaries that the unit is to ignore. In other words, the user may cutpieces out of the defined area that will not be analyzed by the unit, ineffect setting up interior boundaries. This would enable the user todefine exactly the area that is to be analyzed by the unit. Once thearea has been defined the unit then calculated the equation defining theplane 1303. The unit then detects, using edge detection and othermethods known, all lines and curves within the defined area 1304 anddisplays them to the user 1305. The user then deletes all lines that arenot to be modeled 1306. By utilizing the knowledge of the location andorientation of the defined plane in relation to the units position, andthe ability of the unit to measure angles from its bore-sight, by pixelcounting and knowledge of lens zoom state, the unit can calculate thelocation of all of the nodes needed to define those lines and curves inthe defined plane 1308, 1309. FIGS. 9-12 illustrate basic steps involvedin relation to modeling a wall of an office building. In FIG. 9, theuser defines a planar area by ranging to vertices 1-6. Reference withrespect to the users position (which is known to the computer) sets theprecise location of the addressed point in 3 space. In FIG. 10, a userconfirms the process by using an interactive graphical user interface.In FIG. 12 the unit displays the calculated model to the user.

[0038] In FIG. 14, major components of these systems are illustrated ina block diagram. A camera, or cameras 141 are in electricalcommunication with a computer 145. Measurement systems for position 142,attitude 143 and range are arranged together as part of a sensorsubsystem. A video display 146 is also electronically coupled to thecomputer are serves to display images generated therein. Data store 148is also a part of the computer subsystem and contains information suchas computer models both generated and those stored from previoussessions.

[0039] A continuation of the coplanar method would be to use a pair ofplanar models, created from different positions, using the same boundarylimits. If any objects projected from, or were inset into, the plane thetwo images would appear different. The unit would be able to calculate,through triangulation, the correct coordinate for these nodes. Somesituations may require a third view from another location forcompleteness. FIG. 5 shows a general situation in which a user defines aplanar area 51 that has a line 52 sticking out of it at right angles.The viewer would go through the planar modeling process from viewingposition #1 53 and from viewing position #2 54 to generate thecoordinates for node A 55 within the plane and to calculate thecoordinates of the endpoint, node B 56, without the user having to rangeto either point specifically. In this simple situation it would seemeasier to just range on the two points in question, nodes A & B, andhave done with it. But if the area contained many projections orindentations the user, by using this method, would merely have to definea planar area, and view it from several points of view, to model theseprojections in three dimensions. FIG. 6 gives a plan view of thesituation and clearly shows that, if node B 62 is assumed to be in thedefined planar area 61, that the coordinates calculated for theintersection of a line, from the two viewing positions 63,64 to node B62, and the plane 61 will produce different coordinates, nodes B1 66 andB2 65. In the case of node A 61 the change in viewing position will notproduce a change in coordinates because node A 61 is indeed in thedefined plane. These coordinates, as calculated from each viewingposition, are shown in FIGS. 7 and 8. These figures clearly shows thatnode A 71, 81 is within the plane. FIGS. 7 and 8 also show, through thediscrepancies in the positions of modes B1 72 and B2 82, that the realposition of node B is not within the defined plane. To calculate theactual coordinates of node B is quite simple. Given the knowledge of thelocation of each viewing position and the knowledge of the bearings,both horizontal and vertical, to the node, based on the pixel countingangular offset method, from each viewing position the unit couldcalculate the actual position of the node by triangulation.

[0040] The primary elements of the systems are:

[0041] a camera to acquire a digital image of a real scene;

[0042] a computer to run CAD type software and combine graphical objectswith the real image, and further to compute perspective relationshipsbetween a real 3-D scene and a 2-D perspective of that scene, thecomputing routines being responsive to position, attitude and rangingmeasurements; and

[0043] apparatus operable for measuring position, attitude, range.

[0044] More specifically:

[0045] the camera may be a camcorder type, a portable battery operateddevice for producing a digital video signal, in communication with acomputer processor;

[0046] the computer may be the type known as a personal computer: havinga microprocessor for example Pentium or Power PC processors which havebeen employed to execute code developed for computer modeling, furtherbeing operable for combining an image from an electronic camcorder witha computer generated graphical model image, further being incommunication with and responsive to apparatus for measuring position,attitude and range of the camera.

[0047] The apparatus for measuring position, attitude and range may beas follows:

[0048] the position measuring apparatus may be a large scale system suchas the global position system G.P.S. or may be a local system arrangedwith a particular environment in mind such as a city block or singleroom of a building, each employing some radio location measures,depending upon desired accuracy, the choice of appropriate positionmeasuring system will vary;

[0049] the attitude measuring apparatus may be an electronic compass,flux gate compass, interferometer ring gyro or other device which isoperable for measuring relative pointing direction of the camera imagingaxis; and

[0050] the ranging apparatus may be an ultrasonic or radar device whichmeasures relative position of object by timing a test pulse which isreflected therefrom. Some of these devices are common to simplephotographic cameras in auto-focus systems. Other types of ranging, suchas laser or infrared rangefinders, may be equally effective as long asit is possible to determine the distance from the camera to a particularobject and relay that measurement to the computer.

[0051] In one special version of these inventions, cameras are omitted.A viewing path is defined by a reticule system which provides referenceor “pointing” axis for the devices. In common CAD methods, “pointing”and “clicking” may be done via a mouse peripheral device. An icon, forexample an arrow, designates a position on the model field as displayedon the screen. Objects of the model are manipulated by pointing andclicking, clicking and dropping, etc. The model may relate to some realscene, but is not generally responsive thereto.

[0052] In a particular version of the invention, pointing, clicking anddragging operations may be implemented by manipulation of a pointingaxis into a real scene. Clicking is accomplished by a simple switch onthe device sensitive to tactile manipulation. Again, the system'sability to know its position and pointing attitude as well as certainrange data allows it to directly interface with the real world. Themodel is sensitive to and responsive to the position and attitude ofdevice in the 3-D scene. The displayed model, therefore, corresponds tohow the scene would look from any perspective.

[0053] Modeling methods of these inventions include methods having theprimary steps as follows: viewing a scene with an electronic camera;measuring position and attitude of said camera; recording a first pointassociated with said measurements; changing either the position state orattitude state of the camera; recording at least one other pointassociated with the new position and attitude state; and displaying saidpoints superimposed with an image captured with said electronic camera.In this way, one can see the scene while also viewing the model beingformed together in the same perspective. The methods may also include astep where the points are connected together by line segments. Or,further by connecting several points to form a polygon computer modelobject. Such polygons may be shaded, colored, or textured by applicationof these attributes in the computer model databases.

[0054] Where a model of an object in a scene is formed from one point ofview, the user may later go to a different point of view and repeat someor all of the points. From a second position, the points which arerepeated may be averaged with the points acquired from the previousviewpoint to form a more accurate point definition.

[0055] Although the present invention has been described in considerabledetail with clear and concise language and with reference to certainpreferred versions thereof including the best modes anticipated by theinventor, other versions are possible. Therefore, the spirit and scopeof the appended claims should not be limited by the description of thepreferred versions contained therein.

What is claimed is: 1) Apparatus for creating digital computer modelscomprising: at least one electronic camera; position and attitudemeasurement means; a video display; and a computer processor operablefor: running CAD software, acquiring images from said electronic camera,receiving position and attitude information, computing perspectiveadjustments, combining imagery from said electronic camera with imageryfrom said CAD software, displaying combined imagery at said display;said electronic camera, position and attitude measurement means each inelectronic communication with said computer. 2) Apparatus of claim 1further comprising: a range measurement means, said computer furtherbeing operable for receiving range information relating to the distancebetween the apparatus and a point or position in the scene beingaddressed, said range measurement means is in communication with saidcomputer. 3) Apparatus of claim 1, said displaying combined imageryincludes forming a composite image of a real scene with a computer modelgraphic image superimposed thereon in a perspective which corresponds tothe perspective of the scene as viewed from a user's position. 4)Computer modeling apparatus comprised of: a computer processor incommunication with an electronic camera, position, attitude and rangedetermining means; range determining means; and a video display, saidelectronic camera having an imaging axis and an image plane, the imagingaxis defining a system pointing direction, the intersection of theimaging axis and image plane defining a position reference point; saidcomputer arranged to run CAD software in conjunction with softwarearranged to communicate with said position, attitude and rangedetermining means and further with said video display; said positiondetermining means arranged to determine the position of the referencepoint, said attitude determining means arranged to determine the systempointing direction, said range determining means arranged to determinethe distance from the position reference point to a point on an objectin a scene being addressed, and said display having a substantiallyplanar image field with its normal direction aligned to the pointingdirection. 5) A computer apparatus of claim 4, said computer including asoftware object model responsive to position, attitude and range of theapparatus. 6) A computer apparatus of claim 5, said response being atranslation of perspective and size whereby the perspective of the scenebeing addressed from the camera viewpoint corresponds to the perspectiveand size of the displayed model. 7) Computer modeling methods comprisingthe steps: addressing a scene with an electronic camera; measuringposition and attitude of said camera; recording a first point associatedwith said measurements; changing either the position state or attitudestate of the camera; recording at least one other point associated withthe new position and attitude state; and displaying said points recordedsuperimposed with an image captured with said electronic camera. 8)Computer modeling methods of claim 7, further comprising a step tore-acquire a previously defined point or model from a new position toimprove the accuracy by averaging.